Group 3 for MSDS 7331 Lab One: Visualization and Data Preprocessing¶
Analysis of Online News Popularity Dataset (https://archive.ics.uci.edu/ml/datasets/Online+News+Popularity): explore the statistical summaries of the features, visualize the attributes, and make conclusions from the visualizations and analysis
01. Business Understanding (10)¶
- Describe the purpose of the data set you selected (i.e., why was this data collected in the first place?). Describe how you would define and measure the outcomes from the dataset. That is, why is this data important and how do you know if you have mined useful knowledge from the dataset? How would you measure the effectiveness of a good prediction algorithm? Be specific *
This Online News Popularity Dataset was acquisited from Mashable (https://mashable.com) on 01/08/2015 and the goal is to predict the number of shares in social networks (popularity).
Attribute Information:
02. Data Mining Type (10)¶
Describe the meaning and type of data (scale, values, etc.) for each attribute in the data file.
Number of Attributes: 61 (58 predictive attributes, 2 non-predictive, 1 goal field)
url: URL of the article (non-predictive)
- timedelta: Days between the article publication and the dataset acquisition (non-predictive)
- n_tokens_title: Number of words in the title
- n_tokens_content: Number of words in the content
- n_unique_tokens: Rate of unique words in the content
- n_non_stop_words: Rate of non-stop words in the content
- n_non_stop_unique_tokens: Rate of unique non-stop words in the content
- num_hrefs: Number of links
- num_self_hrefs: Number of links to other articles published by Mashable
- num_imgs: Number of images
- num_videos: Number of videos
- average_token_length: Average length of the words in the content
- num_keywords: Number of keywords in the metadata
- data_channel_is_lifestyle: Is data channel 'Lifestyle'?
- data_channel_is_entertainment: Is data channel 'Entertainment'?
- data_channel_is_bus: Is data channel 'Business'?
- data_channel_is_socmed: Is data channel 'Social Media'?
- data_channel_is_tech: Is data channel 'Tech'?
- data_channel_is_world: Is data channel 'World'?
- kw_min_min: Worst keyword (min. shares)
- kw_max_min: Worst keyword (max. shares)
- kw_avg_min: Worst keyword (avg. shares)
- kw_min_max: Best keyword (min. shares)
- kw_max_max: Best keyword (max. shares)
- kw_avg_max: Best keyword (avg. shares)
- kw_min_avg: Avg. keyword (min. shares)
- kw_max_avg: Avg. keyword (max. shares)
- kw_avg_avg: Avg. keyword (avg. shares)
- self_reference_min_shares: Min. shares of referenced articles in Mashable
- self_reference_max_shares: Max. shares of referenced articles in Mashable
- self_reference_avg_sharess: Avg. shares of referenced articles in Mashable
- weekday_is_monday: Was the article published on a Monday?
- weekday_is_tuesday: Was the article published on a Tuesday?
- weekday_is_wednesday: Was the article published on a Wednesday?
- weekday_is_thursday: Was the article published on a Thursday?
- weekday_is_friday: Was the article published on a Friday?
- weekday_is_saturday: Was the article published on a Saturday?
- weekday_is_sunday: Was the article published on a Sunday?
- is_weekend: Was the article published on the weekend?
- LDA_00: Closeness to LDA topic 0
- LDA_01: Closeness to LDA topic 1
- LDA_02: Closeness to LDA topic 2
- LDA_03: Closeness to LDA topic 3
- LDA_04: Closeness to LDA topic 4
- global_subjectivity: Text subjectivity
- global_sentiment_polarity: Text sentiment polarity
- global_rate_positive_words: Rate of positive words in the content
- global_rate_negative_words: Rate of negative words in the content
- rate_positive_words: Rate of positive words among non-neutral tokens
- rate_negative_words: Rate of negative words among non-neutral tokens
- avg_positive_polarity: Avg. polarity of positive words
- min_positive_polarity: Min. polarity of positive words
- max_positive_polarity: Max. polarity of positive words
- avg_negative_polarity: Avg. polarity of negative words
- min_negative_polarity: Min. polarity of negative words
- max_negative_polarity: Max. polarity of negative words
- title_subjectivity: Title subjectivity
- title_sentiment_polarity: Title polarity
- abs_title_subjectivity: Absolute subjectivity level
- abs_title_sentiment_polarity: Absolute polarity level
- shares: Number of shares (target) *
Can use Manisha's code for presentation
03. Data Quality (15)¶
- Verify data quality: Explain any missing values, duplicate data, and outliers. Are those mistakes? How do you deal with these problems? Give justifications for your methods. *
In [1]:
# Import libraries which will be uses for Lab_01 project
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
%matplotlib inline
In [2]:
# Read csv file
df = pd.read_csv('/Users/shanqinggu/Desktop/OnlineNewsPopularity.csv')
df.info()
In [3]:
# Exclude url and timedelta columns, read from n_tokens_title
df = df.loc[:, ' n_tokens_title':]
df.head() # use df.tail() to read from the bottom
Out[3]:
In [4]:
# Combine and make 'channel', will fix the warn contents below
Lifestyle_df=df[df[' data_channel_is_lifestyle']==1]
Lifestyle_df[' Channel']='Lifestyle'
Entertainment_df=df[df[' data_channel_is_entertainment']==1]
Entertainment_df[' Channel']='Entertainment'
Bus_df=df[df[' data_channel_is_bus']==1]
Bus_df[' Channel']='Bus'
Socmed_df=df[df[' data_channel_is_socmed']==1]
Socmed_df[' Channel']='Socmedia'
Tech_df=df[df[' data_channel_is_tech']==1]
Tech_df[' Channel']='Tech'
World_df=df[df[' data_channel_is_world']==1]
World_df[' Channel']='World'
df=pd.concat([Lifestyle_df,Entertainment_df,Bus_df,Socmed_df,Tech_df,World_df],axis=0)
In [5]:
# Combine and make 'weekday', will fix the warn contents below
Monday_df=df[df[' weekday_is_monday']==1]
Monday_df[' weekday']='Monday'
Tuesday_df=df[df[' weekday_is_tuesday']==1]
Tuesday_df[' weekday']='Tuesday'
Wednesday_df=df[df[' weekday_is_wednesday']==1]
Wednesday_df[' weekday']='Wednesday'
Thursday_df=df[df[' weekday_is_thursday']==1]
Thursday_df[' weekday']='Thursday'
Friday_df=df[df[' weekday_is_friday']==1]
Friday_df[' weekday']='Friday'
Saturday_df=df[df[' weekday_is_saturday']==1]
Saturday_df[' weekday']='Saturday'
Sunday_df=df[df[' weekday_is_sunday']==1]
Sunday_df[' weekday']='Sunday'
df=pd.concat([Monday_df,Tuesday_df,Wednesday_df,Thursday_df,Friday_df,Saturday_df,Sunday_df],axis=0)
In [6]:
# Check column location and prepare to drop
df.columns.get_loc(' data_channel_is_lifestyle')
df.columns.get_loc(' data_channel_is_world')
df.columns.get_loc(' weekday_is_monday')
df.columns.get_loc(' is_weekend')
Out[6]:
In [7]:
df.columns[[11, 12, 13, 14, 15, 16, 29, 30, 31,32, 33, 34, 35, 36 ]]
Out[7]:
In [8]:
# Remove previous channel and weekly columns
df.drop(df.columns[[11, 12, 13, 14, 15, 16, 29, 30, 31,32, 33, 34, 35, 36 ]], axis=1, inplace=True)
In [9]:
# No Missing values in this dataset
pd.isnull(df).sum()
Out[9]:
In [10]:
# No duplicated values in this dataset
df[df.duplicated(keep=False)]
Out[10]:
In [ ]:
# Outliers will be handled by log transformation due to the sample numbers are more than 30k
In [11]:
# Find which variables need to do log transform
df_T = df.describe().T
df_T["log"] = (df_T["max"] > df_T["50%"] * 10) & (df_T["max"] > 1)
df_T["log+2"] = df_T["log"] & (df_T["min"] < 0)
df_T["scale"] = ""
df_T.loc[df_T["log"],"scale"] = "log"
df_T.loc[df_T["log+2"],"scale"] = "log+2"
df_T[["mean", "min", "50%", "max", "scale"]]
Out[11]:
In [12]:
# Log transform 18 variables.
df['log_n_tokens_content'] = np.log(df[' n_tokens_content'] + 0.1)
df['log_n_unique_tokens'] = np.log(df[' n_unique_tokens'] + 0.1)
df['log_n_non_stop_words'] = np.log(df[' n_non_stop_words'] + 0.1)
df['log_n_non_stop_unique_tokens'] = np.log(df[' n_non_stop_unique_tokens'] + 0.1)
df['log_num_hrefs'] = np.log(df[' num_hrefs'] + 0.1)
df['log_num_self_hrefs'] = np.log(df[' num_self_hrefs'] + 0.1)
df['log_num_imgs'] = np.log(df[' num_imgs'] + 0.1)
df['log_num_videos'] = np.log(df[' num_videos'] + 0.1)
df['log_kw_min_min'] = np.log(df[' kw_min_min'] + 2)
df['log_kw_max_min'] = np.log(df[' kw_max_min'] + 0.1)
df['log_kw_avg_min'] = np.log(df[' kw_avg_min'] + 2)
df['log_kw_min_max'] = np.log(df[' kw_min_max'] + 0.1)
df['log_kw_max_avg'] = np.log(df[' kw_max_avg'] + 0.1)
df['log_kw_avg_avg'] = np.log(df[' kw_avg_avg'] + 0.1)
df['log_self_reference_min_shares'] = np.log(df[' self_reference_min_shares'] + 0.1)
df['log_self_reference_max_shares'] = np.log(df[' self_reference_max_shares'] + 0.1)
df['log_self_reference_avg_sharess'] = np.log(df[' self_reference_avg_sharess'] + 0.1)
df['log_shares'] = np.log(df[' shares'] + 0.1)
In [ ]:
In [13]:
# find locations for untransformed
df.columns[[1, 2, 3, 4, 5, 6, 7, 8, 11, 12, 13, 14, 18, 19, 20, 21, 22, 44]]
Out[13]:
In [14]:
# Drop the above
df.drop(df.columns[[1, 2, 3, 4, 5, 6, 7, 8, 11, 12, 13, 14, 18, 19, 20, 21, 22, 44]], axis=1, inplace=True)
In [15]:
# Check if everything correct so far
df.dtypes
Out[15]:
In [16]:
# Linear Regression analysis. This part will be moved to # 9. Put here just to show the process to pick interesting variables.
class_y = df.log_shares
class_X = df.drop(['log_shares', ' Channel', ' weekday'], axis=1) # axis = 1 - column
import statsmodels.api as sm
class_X = sm.add_constant(class_X)
ls_model = sm.OLS(class_y.astype(float), class_X.astype(float)).fit()
ls_model.summary()
Out[16]:
!!! Strong multicollinearity
Choose 20 from above and remove problem varibles based on pairplot (not shown)
average_token_length num_keywords kw_max_max kw_min_avg global_subjectivity title_sentiment_polarity abs_title_subjectivity log_n_tokens_content log_n_unique_tokens log_num_hrefs log_num_self_hrefs log_num_imgs log_num_videos log_kw_max_min log_kw_avg_min log_kw_min_max log_kw_avg_avg log_self_reference_min_shares log_self_reference_max_shares log_self_reference_avg_sharess
In [17]:
# Remove some varibles after checking pairplot and keep 12 and target 'log_shares'.
df_clean = df[[' average_token_length', ' num_keywords', ' global_subjectivity',' title_sentiment_polarity',
' abs_title_subjectivity', 'log_n_tokens_content', 'log_n_unique_tokens','log_num_hrefs',
'log_num_self_hrefs','log_num_imgs', 'log_num_videos', 'log_self_reference_avg_sharess', 'log_shares']]
# Recheck linear regression analysis.
clean_y = df_clean.log_shares
clean_X = df_clean.drop(['log_shares'], axis=1) # axis = 1 - column
import statsmodels.api as sm
clean_X = sm.add_constant(clean_X)
clean_ls_model = sm.OLS(clean_y.astype(float), clean_X.astype(float)).fit()
clean_ls_model.summary()
Out[17]:
!!! no multicollinearity now
04. Simple Statistics (10)¶
- Visualize appropriate statistics (e.g., range, mode, mean, median, variance, counts) for a subset of attributes. Describe anything meaningful you found from this or if you found something potentially interesting. Note: You can also use data from other sources for comparison. Explain why the statistics run are meaningful. *
In [18]:
# Quick statistic summary of the data
df.describe().transpose()
Out[18]:
05. Visualize Attributes (15)¶
- Visualize the most interesting attributes (at least 5 attributes, your opinion on what is interesting). Important: Interpret the implications for each visualization. Explain for each attribute why the chosen visualization is appropriate.*
In [19]:
# Most interesting attributes (12 and 1), just the interesting variables
df_clean.hist(figsize=(12,12))
Out[19]:
06. Explore Joint Attributes (15)¶
- Visualize relationships between attributes: Look at the attributes via scatter plots, correlation, cross-tabulation, group-wise averages, etc. as appropriate. Explain any interesting relationships.*
In [ ]:
# Put the pairplot or scatterplot for all the varibles or just mention we did it?
In [20]:
# Pairplot for log transformed variables, as grouped by Channel
sns.pairplot(df, vars=[' average_token_length', ' num_keywords', ' global_subjectivity',' title_sentiment_polarity',
' abs_title_subjectivity', 'log_n_tokens_content', 'log_n_unique_tokens','log_num_hrefs',
'log_num_self_hrefs','log_num_imgs', 'log_num_videos', 'log_self_reference_avg_sharess'], hue=" Channel", palette="husl", size=3)
Out[20]:
In [21]:
# Pairplot for log transformed variables, as grouped by Weekday
sns.pairplot(df, vars=[' average_token_length', ' num_keywords', ' global_subjectivity',' title_sentiment_polarity',
' abs_title_subjectivity', 'log_n_tokens_content', 'log_n_unique_tokens','log_num_hrefs',
'log_num_self_hrefs','log_num_imgs', 'log_num_videos', 'log_self_reference_avg_sharess'], hue=" weekday", palette="husl", size=2)
Out[21]:
07. Explore Attributes and Class (10)¶
- Identify and explain interesting relationships between features and the class you are trying to predict (i.e., relationships with variables and the target classification).*
08. New Features (5)¶
- Are there other features that could be added to the data or created from existing features? Which ones? *
09. Exceptional Work (10)¶
- You have free reign to provide additional analyses. One idea: implement dimensionality reduction, then visualize and interpret the results.*
In [22]:
numeric = [c for i,c in enumerate(df.columns) if df.dtypes[i] in [np.float64, np.int64]]
len(numeric)
# Correlation
cmap = sns.diverging_palette(255, 133, l=60, n=7, as_cmap=True, center="dark")
sns.clustermap(df[numeric].corr(), figsize=(14, 14), cmap=cmap);
In [23]:
# plot the correlation matrix using seaborn
sns.set(style="darkgrid") # one of the many styles to plot using
df_heatmap = plt.subplots(figsize=(10, 10))
sns.heatmap(df.corr(), cmap="BuPu")
Out[23]:
In [24]:
# Cut log_shares into 2 groups (0, 1)
df_cut = df['log_shares_cut'] = pd.qcut(df['log_shares'], 2, labels = ('unpopular', 'popular'))
In [ ]:
# df['log_shares_cut'] = pd.qcut(df['log_shares'], 3, labels = False)
# pd.qcut(range(5), 3, labels=["good","medium","bad"])
In [25]:
# Get 'log_shares' position
df.columns.get_loc('log_shares')
Out[25]:
In [26]:
# Drop the above column
df.drop(df.columns[46], axis=1, inplace=True)
In [27]:
# Samples for pairplot as group by the log_share_cut (0, 1)
sns.pairplot(df, vars = [' average_token_length', ' num_keywords', ' global_subjectivity',' title_sentiment_polarity',
' abs_title_subjectivity', 'log_n_tokens_content', 'log_n_unique_tokens','log_num_hrefs',
'log_num_self_hrefs','log_num_imgs', 'log_num_videos', 'log_self_reference_avg_sharess'], hue = "log_shares_cut", palette="husl", size=2)
Out[27]:
In [28]:
# Pick log transformed variables, transform and prepare for PCA
from sklearn.preprocessing import StandardScaler
features = [' average_token_length', ' num_keywords', ' global_subjectivity',' title_sentiment_polarity',
' abs_title_subjectivity', 'log_n_tokens_content', 'log_n_unique_tokens','log_num_hrefs',
'log_num_self_hrefs','log_num_imgs', 'log_num_videos', 'log_self_reference_avg_sharess']
# Separating out the features
x = df.loc[:, features].values
# Separating out the target
y = df.loc[:,['log_shares_cut']].values
# Standardizing the features
x = StandardScaler().fit_transform(x)
In [29]:
# Try PCA
from sklearn.decomposition import PCA
pca = PCA(n_components=2)
principalComponents = pca.fit_transform(x)
principalDf = pd.DataFrame(data = principalComponents
, columns = ['principal component 1', 'principal component 2'])
In [30]:
# Concat two component and prepare to plot
finalDf = pd.concat([principalDf, df[['log_shares_cut']]], axis = 1)
finalDf.head(10)
Out[30]:
In [31]:
# Plot 2 component PCA
fig = plt.figure(figsize = (6,6))
ax = fig.add_subplot(1,1,1)
ax.set_xlabel('Principal Component 1', fontsize = 15)
ax.set_ylabel('Principal Component 2', fontsize = 15)
ax.set_title('2 component PCA', fontsize = 20)
log_shares_cuts = ['unpopular', 'popular'] # 0 = unpopular, 1 = popular
colors = ['r', 'b']
for log_shares_cut, color in zip(log_shares_cuts, colors):
indicesToKeep = finalDf['log_shares_cut'] == log_shares_cut
ax.scatter(finalDf.loc[indicesToKeep, 'principal component 1']
, finalDf.loc[indicesToKeep, 'principal component 2']
, c = color
, s = 15)
ax.legend(log_shares_cuts)
ax.grid()
